Optimal Control Lecture 9 : Numerical Methods for Deterministic Optimal Control

Lecturer: Marin Kobilarov Consider the general setting where we are minimizing J = φ(x(t f), t f) + t f t 0 L(x(t), u(t), t)dt, (1) subject to: x(t 0) = x 0 , x(t f) and t f free (2) ˙ x(t) = f (x(t), u(t), t) (3) c(x(t), u(t), t) ≤ 0, for all t ∈ [t 0 , t f ] (4) ψ(x(t f), t f) ≤ 0, (5) We will consider three families of numerical methods for solving the optimal control problem: • dynamic programming: i.e. solution to the HJB equations • indirect methods-based on calculus of variations, and Pontryagin's principle • direct methods-based on a finite-dimensional representation, e.g. using a discrete-time formulation or using a parametrized control signal The distinction between indirect and direct methods lies in whether one first derives continuous optimality conditions and then solves them numerically (indirect) as opposed to first discretiz-ing/parametrizing the problem and then solving it numerically (direct). Solving the HJB is equivalent to finding a value function V (x, t) defined over the whole domain X × [t 0 , t f ] where x ∈ X and t ∈ [t 0 , t f ]. The control law is then found through the optimization: u * (x(t), t) = min u H(x(t), u, ∂ x V (x(t), t), t) and constitutes a global solution to the optimal control problem from any starting pair (x(t), t). In general, it is extremely difficult to compute V for any nonlinear system. Various approximation methods are applicable. 1.1 Discrete space-time methods As we already discussed one possible way to solve the HJB equations is to discretize both space and time, convert the problem into a path-finding problem on a graph. Then we can apply the value function expansion methods (value iteration) starting from the goal set backwards. This works for low-order problems with simple dynamics.